feat(elasticsearch): support for composable index templates #15089
Conversation
Co-authored-by: sergio.gomez <[email protected]>
|
Cursor Agent can help with this pull request. Just |
github-actions
bot
added
the
ingestion
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (3.33%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. 📢 Thoughts on this report? Let us know! |
Co-authored-by: sergio.gomez <[email protected]>
sgomezvillamor
changed the title
Co-authored-by: sergio.gomez <[email protected]>
datahub-cyborg
bot
added
the
needs-review
Co-authored-by: sergio.gomez <[email protected]>
…te-ingestion-issue-5643
| ): | ||
| yield mcp.as_workunit() | ||
| except Exception as e: | ||
| logger.debug(f"Unable to fetch composable index templates: {e}") |
There was a problem hiding this comment.
Is it ok to log this as debug?
There was a problem hiding this comment.
definitely not an error as composable index templates is sort of optional
I can set warning
| custom_properties["num_replicas"] = num_replicas | ||
| # 4. Construct and emit properties | ||
| if is_index: | ||
| custom_properties: Dict[str, str] = {} |
There was a problem hiding this comment.
Isn't this the same as _extract_template_custom_properties?
There was a problem hiding this comment.
Similar, but not the same.
In the case of template, it needs to handle different depending on is_composable
datahub-cyborg
bot
added
pending-submitter-merge
and removed
needs-review
…te-ingestion-issue-5643
…te-ingestion-issue-5643
sgomezvillamor
deleted the
cursor/investigate-opensearch-index-template-ingestion-issue-5643
branch
Fix(elasticsearch): Ingest both legacy and composable index templates
Problem:
The Elasticsearch/OpenSearch source connector previously only ingested legacy index templates (using the
get_template()API). Modern OpenSearch (v2.17) and Elasticsearch (7.8+) also support composable index templates, which are accessed via a different API (get_index_template()) and have a distinct data structure. This led to incomplete index template ingestion, where only legacy templates were discovered, or none if only composable templates existed.Solution:
This PR updates the Elasticsearch source to support both types of index templates:
get_workunits_internalmethod now attempts to fetch both legacy and composable index templates._extract_mcpsmethod has been enhanced to correctly parse the metadata (mappings, settings, aliases) from both legacy templates (root-level fields) and composable templates (fields nested undertemplate).Impact:
This bug fix ensures that all index templates (both legacy and composable) are now correctly ingested from OpenSearch and Elasticsearch clusters, providing comprehensive metadata coverage. The changes are backward compatible with older clusters that only use legacy templates.
Related Issue:
CUS-6603
Slack Thread